Simultaneous remote and local control of computer desktop

ABSTRACT

A method for computing includes running a desktop sharing program on a local computer so as to permit a remote user to view, on a remote computer, which is connected to communicate with the local computer via a network, an image of an entire area of a display screen of the local computer and to operate the local computer by remote input via the remote computer. A local window is defined within the display screen of the local computer for running an application under control of a local user. Local inputs made via a local input device of the local computer by the local user are intercepted so that the local inputs do not interfere with operation of the local computer by the remote user. The intercepted inputs are directed to the local window so as to permit the local user to interact with the application independently of the operation of the local computer by the remote user.

FIELD OF THE INVENTION

The present invention relates generally to computer systems andsoftware, and specifically to remote desktop applications.

BACKGROUND OF THE INVENTION

In typical operation, a personal computer supports a single keyboard anda single pointing device. A number of software tools have beendeveloped, however, to allow more than one pointing device to be used incomputer applications, particularly for multi-player games. Examplesinclude CPNMouse, which is available at SourceForge.net, and ManyMouse,available at icculus.org.

Remote desktop programs enable the user of one computer to view thedesktop of another computer and control the functions of that computerremotely (via the Internet or other network). Some remote desktop toolspermit a computer to be controlled simultaneously by a remote user and alocal user. Such tools may, for example, permit both the local andremote users to move a cursor on the local user's computer screen bymanipulating their respective pointing devices, or to enter text viatheir respective keyboards.

Whereas many tools give the remote user full control over all computeroperations, methods have been proposed for limiting the scope of remotecontrol. For example, U.S. Pat. No. 5,758,110, whose disclosure isincorporated herein by reference, describes a method for task-basedapplication sharing, in which a host user designates an application tobe shared with another user at a remote location, referred to as theclient user. The shared application runs and executes only on the hostsystem. The client system renders an image of all windows of the sharedapplication without also displaying unshared applications. Both theclient and the host users continue to perform normal operations outsideof the shared area, and the host user defines the tasks which are to beshared.

SUMMARY OF THE INVENTION

In embodiments of the present invention, a desktop sharing programrunning on a local computer enables a remote user to view an image ofthe entire area of the display screen of the local computer and tooperate the local computer by remote input. To permit the local user tointeract with the local computer while the remote user is working, alocal window is defined within the display screen of the local computer.The local window can be used to run an application under control of thelocal user independently of the operations performed by the remote user.For this purpose, the local computer intercepts local inputs made by thelocal user and directs the intercepted inputs to the local window, thusenabling the local user to interact with the application withoutinterfering with the work of the remote user.

There is therefore provided, in accordance with an embodiment of thepresent invention, a method for computing, including:

running a desktop sharing program on a local computer so as to permit aremote user to view, on a remote computer, which is connected tocommunicate with the local computer via a network, an image of an entirearea of a display screen of the local computer and to operate the localcomputer by remote input via the remote computer;

defining a local window within the display screen of the local computerfor running an application under control of a local user;

intercepting local inputs made via a local input device of the localcomputer by the local user so that the local inputs do not interferewith operation of the local computer by the remote user; and

directing the intercepted inputs to the local window so as to permit thelocal user to interact with the application independently of theoperation of the local computer by the remote user.

In disclosed embodiments, the local window is not reproduced in theimage of the entire area of the display screen that is presented on theremote computer. Typically, the local window is defined as an overlaywindow on the local computer. Additionally or alternatively, definingthe local window includes displaying the local window as asemi-transparent overlay on the display screen that is presented on thelocal computer.

In some embodiments, running the remote desktop program includes movinga first cursor on the display screen in response to manipulation of aremote pointing device of the remote computer by the remote user, anddirecting the intercepted inputs includes moving a second cursor on thedisplay screen, independently of the first cursor, under the control ofthe local user by means of a local pointing device of the localcomputer. The second cursor may be effective to select objects on thedisplay screen only within the local window.

Additionally or alternatively, intercepting the local inputs includesintercepting keystrokes made by the local user by means of a localkeyboard of the local computer, and directing the intercepted inputsincludes displaying input text, responsively to the interceptedkeystrokes, in the local window. In one embodiment, running theapplication includes running a textual chat application, in which theinput text is transmitted over the network to the remote user, andtextual responses from the remote user are displayed in the localwindow. The desktop sharing program may be used by the remote user inproviding service support to the local user.

There is also provided in accordance with an embodiment of the presentinvention, a computer software product, including a tangiblecomputer-readable medium in which program instructions are stored, whichinstructions, when read by a local computer, cause the local computer torun a desktop sharing program so as to permit a remote user to view, ona remote computer, which is connected to communicate with the localcomputer via a network, an image of an entire area of a display screenof the local computer and to operate the local computer by remote inputvia the remote computer,

wherein the instructions cause the local computer to define a localwindow within the display screen of the local computer for running anapplication under control of a local user, to intercept local inputsmade via a local input device of the local computer by the local user sothat the local inputs do not interfere with operation of the localcomputer by the remote user, and to direct the intercepted inputs to thelocal window so as to permit the local user to interact with theapplication independently of the operation of the local computer by theremote user.

There is additionally provided, in accordance with an embodiment of thepresent invention, a local computer, including:

a local input device;

a display monitor; and

a processor, which is configured to run a desktop sharing program so asto permit a remote user to view, on a remote computer, which isconnected to communicate with the local computer via a network, an imageof an entire area of a display screen of the local computer and tooperate the local computer by remote input via the remote computer, andto define a local window within the display screen that is displayed onthe display monitor of the local computer for running an applicationunder control of a local user, to intercept local inputs made via thelocal input device by the local user so that the local inputs do notinterfere with operation of the local computer by the remote user, andto direct the intercepted inputs to the local window so as to permit thelocal user to interact with the application independently of theoperation of the local computer by the remote user.

The present invention will be more fully understood from the followingdetailed description of the embodiments thereof, taken together with thedrawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic, pictorial illustration of a system providingremote control of a local computer, in accordance with an embodiment ofthe present invention;

FIGS. 2A, 2B and 2C are schematic representations of display screens ofa local computer and a remote computer, in accordance with an embodimentof the present invention;

FIG. 3 is a block diagram that schematically shows functional componentsof local and remote computers, in accordance with an embodiment of thepresent invention; and

DETAILED DESCRIPTION OF EMBODIMENTS Overview

Although remote desktop programs commonly allow both the remote user andthe local user to interact with the computer via their respective inputdevices, this situation may lead to conflicts. For example, the localand remote users may simultaneously try to move the onscreen cursorusing their respective pointing devices (sometimes referred to as“fighting over the mouse”), or

to enter text input in an application window using their keyboards.

One sort of situation in which conflicts of this sort may arise is inremote support applications, in which the remote user services the localuser's computer and/or instructs the local user in proper operation ofthe computer. In such situations, it is often desirable for the localand remote users to conduct an on-screen textual chat while the remoteuser is working on the local user's computer. If both the remote userand the local user attempt to enter text in the chat window at the sametime, however, the result may be at best annoying and possiblyunintelligible to both users. (This is one reason why in currentpractice, it is common for the remote user to keep a telephoneconnection open with the local user during such support sessions.)Problems may similarly arise if both the remote user and the local userattempt to move the on-screen cursor using their respective pointingdevices.

The embodiments of the present invention that are described hereinbelowpermit the remote user to exercise full control over the functions ofthe local computer, and to view the entire area of the local computerdisplay screen, while still leaving a local window under the control ofthe local user. Typically, the local window appears only on the localcomputer screen and is not reproduced in the image of the screen that isrendered on the remote computer. A desktop sharing engine on the localcomputer intercepts local inputs made by the local user via local inputdevices, such as the keyboard and mouse, and directs these inputs to thelocal window, while permitting the remote user to control all otheroperations of the local computer by input from the remote computer.

The local window is particularly useful in supporting textual chatbetween the local and remote users, in the sort of support applicationsthat are described above, for example, but it can alternatively be usedto run substantially any sort of application on the local computer (forexample, a game of Solitaire to pass the time until the remote user hasfinished working). When the local window is active, two cursors mayappear on the local computer screen: the system cursor, controlled bythe remote user's pointing device, and a local cursor, controlled by thelocal input that is intercepted by the desktop sharing engine. Bothcursors may be used by their respective users simultaneously andindependently to select objects on the screen, including performingoperations such as drag/drop and cut/paste.

Various implementations of the principles of the present invention arepossible, depending on user requirements and system constraints. In theembodiments described hereinbelow, a single local window is used fortextual chat in a remote support application, and the local cursor maybe limited to operating in this window. In other embodiments, multiplelocal windows may be deployed, with configurable properties, and thelocal cursor may have extended capabilities. For example, the localwindows may be accessible only to the local user or may be shared withthe remote user, and the local user may be prompted to indicate, uponopening a local window, whether or not this window should be accessibleto the remote user. The local user may be required to selectapplications to open in a local window from a “private” applicationlist, in order to avoid fighting over the mouse when attempting to openand control the application.

System Description

FIG. 1 is a schematic, pictorial illustration of a system 20 forproviding remote control of a local computer 22, in accordance with anembodiment of the present invention. The local computer, which isoperated by a local user 24, communicates via a network 30, such as theInternet or a local area network (LAN), with a remote computer 26,operated by a remote user 28. The terms “local” and “remote” are usedherein arbitrarily, for the sake of convenience and clarity inidentifying the computers that are involved in system 20 and theirrespective users and peripheral devices. “Local” refers to the computerwhose operations are controlled by another computer, while “remote”refers to the controlling computer. There is not necessarily anysubstantive difference between the computers themselves, however, andthe roles of the local and remote computers could just as well bereversed by appropriate software configuration.

Local computer 22 and remote computer 26 are assumed to comprisepersonal computers, with appropriate peripheral devices and software forcarrying out the functions described herein. Alternatively, one or bothof the computers may comprise a server or any other suitable type offixed or portable computing device with an appropriate communicationlink and software. In the present embodiment, local computer 22comprises a processor 32, with a display monitor 34 and input devices,such as a keyboard 36 and a mouse 38. Remote computer 26 comprisessimilar components: a processor 40, display monitor 42, keyboard 44 andmouse 46. Alternatively or additionally, one or both of the computersmay comprise other sorts of peripheral devices, such as a joystick,trackball, or other type of pointing device (in addition to or insteadof the mouse). Although certain features of system 20 are describedhereinbelow with specific reference to mouse 38 and mouse 46, thesefeatures are equally applicable to other types of pointing devices.

The embodiments that are described below rely on novel desktop sharingsoftware that runs on processors 32 and 40. This software may bedownloaded to computers 22 and 26 in electronic form, over a network,for example, or it may additionally or alternatively be provided ontangible storage media, such as optical, magnetic, or electronic memorymedia. In the description that follows, it is assumed that the desktopsharing software runs in the environment of a standard operating system,such as a Microsoft® Windows® operating system, and some features of thesoftware are described, by way of example, with reference to featuresand components of Windows. The principles of the present invention,however, may likewise be applied, mutatis mutandis, in other sorts ofoperating environments, such as UNIX®, Linux®, or Apple® operatingenvironments, as well as in portable computing and communicationdevices, such as personal digital assistants (PDA's) and smart phones.

FIGS. 2A, 2B and 2C are schematic representations of display screens 47,48 and 49, in accordance with an embodiment of the present invention.Screens 47 and 48 are displayed on monitor 34 of local computer 22,while screen 49 is displayed on monitor 42 of remote computer 26 insystem 20.

In the example scenario illustrated in FIGS. 2A-2C local user 24, namedTom, uses an Internet browser (window 52) to connect to a support Website, which offers expert services in fixing computer problemsencountered by the local user. (Alternatively, Tom may connect to thesupport site using other types of applications and communicationprotocols.) As shown in FIG. 2A, a window 47 is displayed by the browseron the local computer, showing entries 51 offering support services by anumber of experts. In this example, the local user selects an expert,named Joe, on the Web site. This expert takes control of local computer22 (after receiving approval from Tom) as remote user 28 of computer 26,using desktop sharing software that is installed on both of thecomputers. As a results, screens 48 and 49 (FIGS. 2B and 2C) aredisplayed on the local and remote computers, respectively.

Screen 48 (commonly referred to as the “desktop” in the Windowsparadigm), as seen by local user 24, includes application windows 50 and52, as well as a system cursor 54, which are defined and controlled bythe operating system of computer 22. A local window 56, with a localcursor 58, is overlaid on screen 48, with novel functionality asdescribed hereinbelow. The background of window 56 may be madesemi-transparent, by appropriately setting the transparency attributeprovided by the Windows operating system, in order to enable local user24 to view the parts of screen 48 that are below the local window. (Forclarity of illustration, however, this feature of window 56 is not shownin FIG. 2A.)

Screen 49, as seen by remote user 28, contains an image 60 of the entirearea of screen 48, including windows 50 and 52 and cursor 54. The remoteuser is able to manipulate cursor 54 using mouse 46, as well asinputting text in windows 50 and 52 via keyboard 44, in the manner thatis known generally in the art of remote desktop computing. The remoteuser can thus interact with and control functions of the operatingsystem and application software running on local computer 22. Localwindow 56 and cursor 58, however, are typically not reproduced on screen49, since these elements are reserved for the use of local user 24.

While Joe is working on local computer 22, a textual chat applicationruns on computer 22 in local window 56. Tom inputs text to the chat inwindow 56 using keyboard 36, and may also select objects on screen andmove them into or within window 56 using cursor 58, under the control ofmouse 38. Text that Tom inputs into window 56 appears in a correspondingchat window 62 on screen 49. Joe may type his text responses into thiswindow, whereupon these responses are presented to Tom in window 56, asillustrated in the figures.

As noted earlier, this sort of textual chat is a particularly usefulapplication of local window 56. In the absence of such a dedicated localwindow, the local and remote users could take turns inputting text tothe same window, using their respective keyboards, but this sort ofshared control is prone to collision between the respective inputs andoften results in confusion and frustration to the users. On the otherhand, the local window on computer 22 may be used not only for textualchat, but also for any other sort of application that the local user maywish to run (though applications that could interfere with the work ofthe remote user are probably best avoided).

Software Structure and Operation

FIG. 3 is a block diagram that schematically shows functional componentsof computers 22 and 26 that are involved in the present method ofdesktop sharing, in accordance with an embodiment of the presentinvention. A local desktop sharing engine 66 runs on local computer 22,while a remote desktop sharing engine 68 runs on remote computer 26. Thedesktop sharing engines may be based, for example, on UltraVNC RemoteControl Software (which is available on line at uvnc.com), with certainadded features in local engine 66 for supporting local window 56 andcursor 58, as described hereinbelow.

Remote engine 68 receives information from local engine 66 defining thewindows on screen 48 and their contents, and uses this information inrendering image 60 on the screen of computer 26. In the MicrosoftWindows environment, for example, the information may be provided usingthe application programming interface (API) of the Windows GraphicsDevice Interface (GDI) subsystem. Engine 68 also captures inputs made byremote user 28 using keyboard 44 and mouse 46 and transmits these inputsvia network 30 to engine 66, as well as displaying the result of theinputs in image 60. In response to the inputs sent from remote engine68, local engine 66 drives cursor 54 and controls functions of localcomputer 22 as invoked by the remote user.

Before transmitting screen information to remote engine 68, local engine66 may filter out the information relating to local window 56. As aresult, the remote engine will not reproduce window 56 in image 60, asillustrated in FIG. 2B. Such filtering may be achieved, if local engine66 uses UltraVNC, simply by defining window 56 as semi-transparent,because UltraVNC as currently configured does not capture GDIinformation regarding semi-transparent overlay windows. The image screencapture feature of UltraVNC should also be turned off.

Another way to filter out the local window, when the local window isdefined as an overlay window, is to use the Bi tBl t function of the GDIAPI to capture images of screen 48 both with and without the CAPTUREBLTflag enabled. When CAPTUREBLT is enabled, local engine 66 captures animage of the screen that includes overlay windows (such as window 56)When it is disabled, the overlay windows are omitted from the capture.The local engine combines the images in order to generate an image ofscreen 48 from which window 56 has been excised, while revealing anywindows (such as window 50 in FIG. 2A) underneath window 56.

There is a block diagram that schematically illustrates softwarecomponents on local computer 22 that are involved in operating cursors54 and 58, in accordance with an embodiment of the present invention. Innormal Windows operation, a mouse driver captures signals from mouse 38and generates events (messages) representing movement of the mouse anduser clicks on the mouse buttons. A mouse class driver receives andpasses these events to a system input queue which then places the eventsin a raw input thread (RIT) queue. From this queue, the events arepassed as WM_* mouse messages to respective virtual input queues ofapplication threads, which generate the appropriate actions of cursor 54on screen 48.

When computer 22 is under remote control by computer 26, as describedabove, local desktop sharing engine 66 receives mouse-related events vianetwork 30 from remote engine 68 and places these events directly in RITqueue. The remote events bypass driver and queue, and thus permit remoteuser 28 to manipulate cursor 54. This sort of remote cursor control isimplemented, for example, as part of the standard UltraVNC softwarepackage described above. Keyboard events on computer 26 are likewisetransmitted to and inserted by local engine 66 in the appropriate inputqueue.

While computer 22 is under remote control, a filter intercepts eventsfrom driver that correspond to local inputs by user 24 using mouse 38.These local events are not permitted to pass to mouse class driver,since otherwise they would be enqueued and passed to applicationthreads, thus “competing” with the remote input for control of cursor54.

Instead, filter passes the events to local desktop sharing engine 66,which generates “faked” local cursor 58. The cursor is “faked” in thesense that it is not generated by the operating system of computer 22,but is rather rendered on screen 48 by engine 66 in a manner thatemulates the operation of “real” cursor 54. Typically, the cursoremulation performed by engine 66 enables local user 22 to use cursor 58both to point to particular objects (such as words, blocks of text,icons, or other images) and to select and even move objects on screen.Thus, cursor 58 may be used to copy and paste or to drag and drop ablock of text or an image,

independently of (and without interfering with) similar operationsperformed by remote user 26 using cursor 54, even when the operationsare carried out by both users simultaneously.

Since cursor 58 is generated at the application level, local engine 66may be configured to limit the range of objects on which the cursor isable to operate. For example, cursor 58 may be tied to the applicationrunning in window 56 and may thus be limited to manipulating objectsappearing in this window. Alternatively, local engine 66 may beconfigured to permit cursor 58 to operate on objects outside window 56,although in any case it is desirable to limit the functionality ofcursor 58 so as not to interfere with the operations carried out byremote user 28.

A similar sort of filtering arrangement (not shown in the figures) maybe used to intercept keystroke input made by local user 24 via keyboard36 and/or inputs via other local input devices. Local engine 66processes the keystrokes so as to generate text for display in window56.

Although the embodiments described hereinabove relate to a particularapplication context, the principles of the present invention maysimilarly be applied in substantially any sort of desktop sharingscenario in which multiple users wish to view and operate on the samescreen simultaneously. It will thus be appreciated that the embodimentsdescribed above are cited by way of example, and that the presentinvention is not limited to what has been particularly shown anddescribed hereinabove. Rather, the scope of the present inventionincludes both combinations and subcombinations of the various featuresdescribed hereinabove, as well as variations and modifications thereofwhich would occur to persons skilled in the art upon reading theforegoing description and which are not disclosed in the prior art.

1. A method for computing, comprising: running a desktop sharing programon a local computer so as to permit a remote user to view, on a remotecomputer, which is connected to communicate with the local computer viaa network, an image of an entire area of a display screen of the localcomputer and to operate the local computer by remote input via theremote computer; defining a local window within the display screen ofthe local computer for running an application under control of a localuser; intercepting local inputs made via a local input device of thelocal computer by the local user so that the local inputs do notinterfere with operation of the local computer by the remote user; anddirecting the intercepted inputs to the local window so as to permit thelocal user to interact with the application independently of theoperation of the local computer by the remote user.
 2. The methodaccording to claim 1, wherein the local window is not reproduced in theimage of the entire area of the display screen that is presented on theremote computer.
 3. The method according to claim 2, wherein the localwindow is defined as an overlay window on the local computer.
 4. Themethod according to claim 2, wherein defining the local window comprisesdisplaying the local window as a semi-transparent overlay on the displayscreen that is presented on the local computer.
 5. The method accordingto claim 1, wherein running the remote desktop program comprises movinga first cursor on the display screen in response to manipulation of aremote pointing device of the remote computer by the remote user, andwherein directing the intercepted inputs comprises moving a secondcursor on the display screen, independently of the first cursor, underthe control of the local user by means of a local pointing device of thelocal computer.
 6. The method according to claim 5, wherein the secondcursor is effective to select objects on the display screen only withinthe local window.
 7. The method according to claim 1, whereinintercepting the local inputs comprises intercepting keystrokes made bythe local user by means of a local keyboard of the local computer, andwherein directing the intercepted inputs comprises displaying inputtext, responsively to the intercepted keystrokes, in the local window.8. The method according to claim 7, wherein running the applicationcomprises running a textual chat application, in which the input text istransmitted over the network to the remote user, and textual responsesfrom the remote user are displayed in the local window.
 9. The methodaccording to claim 8, wherein the desktop sharing program is used by theremote user in providing service support to the local user.
 10. Acomputer software product, comprising a tangible computer-readablemedium in which program instructions are stored, which instructions,when read by a local computer, cause the local computer to run a desktopsharing program so as to permit a remote user to view, on a remotecomputer, which is connected to communicate with the local computer viaa network, an image of an entire area of a display screen of the localcomputer and to operate the local computer by remote input via theremote computer, wherein the instructions cause the local computer todefine a local window within the display screen of the local computerfor running an application under control of a local user, to interceptlocal inputs made via a local input device of the local computer by thelocal user so that the local inputs do not interfere with operation ofthe local computer by the remote user, and to direct the interceptedinputs to the local window so as to permit the local user to interactwith the application independently of the operation of the localcomputer by the remote user.
 11. The product according to claim 10,wherein the local window is not reproduced in the image of the entirearea of the display screen that is presented on the remote computer. 12.The product according to claim 11, wherein the local window is definedas an overlay window on the local computer.
 13. The product according toclaim 11, wherein the instructions cause the local computer to displaythe local window as a semi-transparent overlay on the display screenthat is presented on the local computer.
 14. The product according toclaim 10, wherein the instructions cause the local computer to move afirst cursor on the display screen in response to manipulation of aremote pointing device of the remote computer by the remote user, and tomove a second cursor on the display screen, independently of the firstcursor, under the control of the local user by means of a local pointingdevice of the local computer.
 15. The product according to claim 14,wherein the second cursor is effective to select objects on the displayscreen only within the local window.
 16. The product according to claim10, wherein the instructions cause the local computer to interceptkeystrokes made by the local user by means of a local keyboard of thelocal computer, and to display input text, responsively to theintercepted keystrokes, in the local window.
 17. The product accordingto claim 16, wherein the instructions cause the local computer to run atextual chat application, in which the input text is transmitted overthe network to the remote user, and textual responses from the remoteuser are displayed in the local window.
 18. The product according toclaim 17, wherein the desktop sharing program is used by the remote userin providing service support to the local user.
 19. A local computer,comprising: a local input device; a display monitor; and a processor,which is configured to run a desktop sharing program so as to permit aremote user to view, on a remote computer, which is connected tocommunicate with the local computer via a network, an image of an entirearea of a display screen of the local computer and to operate the localcomputer by remote input via the remote computer, and to define a localwindow within the display screen that is displayed on the displaymonitor of the local computer for running an application under controlof a local user, to intercept local inputs made via the local inputdevice by the local user so that the local inputs do not interfere withoperation of the local computer by the remote user, and to direct theintercepted inputs to the local window so as to permit the local user tointeract with the application independently of the operation of thelocal computer by the remote user.
 20. The computer according to claim19, wherein the local window is not reproduced in the image of theentire area of the display screen that is presented on the remotecomputer.
 21. The computer according to claim 19, wherein the localinput device comprises a local pointing device, and wherein theprocessor is configured to move a first cursor on the display screen inresponse to manipulation of a remote pointing device of the remotecomputer by the remote user, and to move a second cursor on the displayscreen, independently of the first cursor, under the control of thelocal user by means of the local pointing device.